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A COMPUTER SOFTWARE SYSTEM FOR ADAPTIVE 
ABILITY MEASUREMENT 



Adaptive , computer-control Led , ability test administration 
has considerable promise for improving ability men suremen t • 
Research to date (Weiss and Betz, 1973) indicates that adap- 
tive tests can yield ability estimates which are more precise 
than those derivable from conventional tests, in addition to 
providing scores with higher validity. Adaptive tests can 
achieve reliabilities and validities at least as high as those 
of conventional ability tests with considerably fewer test 
items, thus saving time on the part of the testee. Such time 
savings can be put to use in the? measurement of other abilities 
to supplement information available on each individual. 

Computerized ability measurement also has other potential 
advantages over paper and pencil testing procedures ( Weiss and 
Betz, 1973, PP« 56-57). Among these are the elimination of 
time limits, thus reducing artif actual speed components in tost, 
scores; improved standardization of test aclminis tra t ion condi- 
tions, thus reducing administrator effects; the possibility of 
measuring new content and modes of abilities not now measurable 
with paper and pencil tests; development of tests measuring 
problem-solving abilities using "within-item" branching as 
opposed to the more typical "between-item" branching; compu- 
terized control of confidence -weighting schemes of response, 
including techniques for differential weighting of response 
distractors; use of item response latency data as additional 
diagnostic (and perhaps predictive) information derivable from 
ability testing; the a priori control of the degree of preci- 
sion attached to any ability estimate derived from ability test; 
administration 5 the tailoring of test difficulty to the indi- 
vidual's ability level, potentially reducing test-taking anxiety 
and increasing test-taking motivation; and the use of knowledge 
of results or feedback after each item is answered (with vary- 
ing ratios of positive to negative reinf orcoment) potentially 
increasing test-taking motivation. 

Should computerized ability testing prove capable of pt^o- 
viding at least some of these potential improvements in measur- 
ing individual differences in abilities, operational computerized 
~a!5Tl~xty testing sy~slf ern s~~wl 1 1 b e developed in the next decade • 
Such systems will find their optimal use in institutions which 
serve populations of wide-ranging ability, and in applications 
where it is desired to measure a large number of abilities in 
a minimum amount of testing time. Thus, military placement 
and selection activities, industrial personnel departments, 
vocational- technical schools, vocational guidance departments 
of high schools and colleges, and public and private vocational 
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counseling agencies are likely users of computerized ability 
tests. In the majority of cases these agencies will simply 
purchase or lease time-shared computer terminal devices, such 
as cathode-ray-terminals (CRTs), or hard-copy devices such as 
medium-speed teletype terminals , for interfaces between the 
testee and the computer. Computer programs to administer 
ability tests will probably be maintained by centralized faci- 
lities which are accessed by conventional telephones linked to 
the central time-shared computer. Just as there are currently 
different testing agencies providing a variety of conventional 
paper and pencil tests for these same test users, it is likely 
that different computer software systems will be available for 
computerized ability testing. These systems will differ not 
only in the strategies of testing available but also in the 
item pools available for ability measurement (both nature and 
quality) and in other system characteristics which allow for 
user flexibility and control. 

Although adaptive ability testing has generated a small 
amount of research to date, very little of the research has 
actually used computer-controlled test administration. Yet 
computerized ability testing raises a host of new questions 
waiting to be addressed by the psychometric research community. 
These problems include (Weiss and Betz, 1973* PP • 49-5*0 com- 
parisons of the almost endless variety of adaptive testing 
procedures possible under computer-controlled administration; 
empirical investigation of the characteristics of the various 
scoring methods within each of the adaptive strategies; new 
problems and approaches to item P°jbl development for compu- 
terized testing; the effects of chance success on the scores 
derived from adaptive test administration; the variety of 
termination rules possible in some adaptive strategies; appli- 
cations of on-line differential weighting procedures; the 
actual effects on the testee of adaptive test administration, 
in terms of reducing anxiety and increasing motivation; and 
the effects on test scores of immediate knowledge of results . 

There is likely, then, to be a demand in the near future 
for computerized ability testing systems for both research 
and applied testing purposes. The present authors have developed 
such a system whic h is op erational and which has logged over 
2,500 hours of actual testing. This report describes the design 
of that system and its rationale, so that it might serve as a 
starting point for others who in the future might wish to develop 
computerized ability testing systems for research and/or applied 
uses. It is hoped that future systems wil 1 improve upon the 
present software system so that computerized ability testing 
will reach its maximum potential. 
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Requirements for a Computerized Ability Testing System 



A minimal system must l) select a test item from an item 
pool following a predetermined set of instructions, 2) present 
the item on a CRT or other display unit, 3) accept a response, 
k) check it for validity and compatibility with the instruc- 
tional sequence, 5) determine whether the response is correct 
or incorrect, 6) calculate response latency, 7) save the ob- 
tained data , and 8) select the next item to be administered 
in accordance with a specified testing strategy. In addition, 
previous experience with computerized ability testing and plans 
guiding the present research dictated a number of features 
which would be desirable in any computerized testing system, 
for either research or applied use. 

One primary characteristic is operational simplicity in 
terms of supervision and t estee/sy stem interaction. This 
dictates several important features of the system: 

1. The system must provide for monitoring by proctors 
with little or no computer experience. Thus, the 
proctor should know simply how to dial the telephone, 
call the testing program, assign testing conditions 
to the testee, and handle common hardware (equipment) 
or software (program) problems as they arise, 

2. The system must be self-instructional • Thus, the pro- 
gram must present a series of instructional screens 
and enforce performance to a criterion on each screen. 
Should the testee not master the instructions for use 
of the equipment after a given number of trials, the 
proctor should be called to assist the testee, 

3. Since testees will occasionally wish to change an answer 
typed to a test item, because of either typing errors 

or actual changes in their answer to an item, the system 
must provide some opportunity to change responses, at 
least before the next item is presented, 

k. In acoustically coupled time-shared systems, data are 
occasionally lost in transmission between the computer 
and the testing terminal (e.g., CRT), This results in 
either the computer being unable to interpret the re- 

s p on s e a ~ t e s t e e~ha s~~ mad e — t o~*a^t es t"it em^ crr~~±n~a~~d i~s~= 

played test item which appears as garbled information • 
In either case, the system must be able to present any 
test item over again if data should be lost in trans- 
mission. Implementation of this feature should not 
require elaborate instructions , 
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A computer-controlled ability testing system used for 
either research or applied testing must be flexible. Thus, 
the following features are important: 

5 . When tes t s are administered for research purposes , 
the only tangible reward for the testee may be know- 
ledge of "how he did" on the test. Thus, a compu- 
terized ability testing system designed for research 
should give test scores at the end of the t es t , norma- 
tive comparisons (if desired by the administrator) 
and an opportunity for the testee to examine his per- 
formance in detail--on an item-by-item baais--under some 
circumstances . This feature should operate following 
completion of testing, with no necessity for inter- 
vention by the proctor. 

6. All computer systems fail occasionally. Failures can 
result from interruption on the telephone lines, power 
failures, failure of the computer's central processor , 
peripheral processor or disks, software failures, or 
failure of terminal equipment. With the large number 
of system components subject to failure, any computer- 
ized testing system must be capable of recovery with 
as little loss of data as possible. Thus, when ab- 
normal terminat ions occur, the proctor should be able 
to re-start the testee at the item on which termina- 
tion occurred once the system is again operational. 
This "recovery" feature would also be important where 
the testee or proctor interrupt s testing for some 
reason (e.g., testee becomes ill , needs to keep an 
appointment elsewhere) , and it is desired to continue 
testing at a later time. 

7. On occasion it might be desirable to terminate a testee 
at some point prior to normal termination, not allow- 
ing for eventual re-entry at the point of termination. 
In this case , the proctor would wish to move directly 
to the post-testing de-briefing routine, including 
total scores and potential review of performance for 
the testee . 

8. Thex^e are many strategies of computerized ability test- 
ing (Weiss and Betz, 1973, PP. 15-^0; Weiss, 1974). 
The^se"i"n"c"Tade" two-stage , pyramidal, "f 1" exiTe v e 1T 9 stFa"- " 
daptive and Bayesian strategies, among others, with 
new ones yet to be developed. For research purposes , 
it is desirable to be able to choose a strategy for 
administration to a given testee at the time he presents 
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himself for testing. For applied testing purposes, it; 
might be discovered that certain kinds of strategies 
are appropriate for certain applied measurement situa- 
tions. Thus, an operational computerized ability test- 
ing system must permit the proctor to select , at the 
time of testing, one of a number of adaptive testing 
strategies for administration to a given testee. At 
the same time , different testees being tested simul- 
taneously on different terminals should be able to be 
administered different strategies . 

9 . Both applied and research adaptive measurement systems 
should concern themselves with different ability do- 
mains. Thus, the computer system should have several 
item "pools" consisting of, for example, items measur- 
ing verbal comprehension , numerical reasoning , cleric a I. 
ability', and spatial ability. Any adaptive testing 
strategy should be able to access any of these pools, 
with the nature of the pool (i.e.', the ability being 
measured) selected at the point of test administration. 
Needless to say, the system should also permit the use 
of different pools with different testing strategies 
simultaneously. 

10. Given an item pool (e.g., vocabulary or number series) 
and a single testing strategy, different subsets of 
items should be available Tor presentation to different 
testees. For example, it might be appropriate to 
administer to one testee a 15-stage pyramidaL voca- 
bulary test and at the same time administer a 
pyramidal vocabulary test to another testee. Further*, 
the 15-stage test might use a fixed step size while 
the 10-stage test used a "shrinking 11 step size. In 

th Ls case, one item subset would be comprised of I lie 
item numbers pre-selected to form the 15-stage fixed 
step size pyramid, while the other item subset would 
include a different set of items structured into a lo- 
st age pyramid. The item subset to be used for a given 
testee, nested within the total vocabulary item pooL 
and administered under a specified adaptive testing 
strategy, would be selected at the point of test ad- 
ministration . 

11. 'Some - st ra t egi e s of " ad ap t iv e~ab ili t y ™t es t ing- (~e-. gvy " 

stradaptive , Bay esian) require an initial estimate of 
ability to determine a starting point for the test 
ad minis trat ion branching procedure . When these s tra - 
tegies are to be administered, the system must request 
the appropriate information from the testee (e.g., his 
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cumulative grade point average or his subjective 
estimate of his standing on the trait in deciles; 
see Weiss, 1973* PP« 12-15) or obtain it from pre- 
viously stored data on the testee. 

Certain other features are dictated by the requirements 
of the research design of the present research: 

12. Research in adaptive testing should be concerned with 
studies of the inter-relationships among ability 
estimates derived from the various testing strategies. 
Thus , it is desirable to administer two or more tests 
to a given testee using different testing strategies. 
At the same time, it might be undesirable for the 
testee to know that he is completing one testing 
strategy and beginning another , when the same item 
pool is being used. Conversely, when two different 
item pools are being used for the same tester in one 
test administration (e.g., vocabulary and numerical 
reasoning) it would be necessary to interpose additional 
instructions between the two tests. Thus, the system 
must allow for smooth and uninterrupted flow between 
two tests within the same item pool, yet permit struc- 
tured movement between item pools. In addition, it 
should permit the administration of only one test when 
multiple tests are not desired. 

13. When multiple tests are desired in one test administra- 
tion period, the system should permit tests to be ad- 
ministered in any order. Such flexibility will permit 
the analysis of, and control for, order effects in the 
administration of various testing strategies. 

14. Research in adaptive testing should also be concerned 
with the test-retest stability of the sc ores obtained 
from various testing strategies. If the testee is 
taking a re test , he should be given the option of re- 
viewing the instructions. Further, if an initial ability 
estimate is needed for the retest, the estimate from 

the previous administration of that test must be re- 
trieved from a data file. This is necessary to ensure 
comparability of test and retest results. 

Several features of the software system are also important 
to the programmer: 

15. It is obvious that new strategies of computerized 
ability testing are yet to be developed. In order to 
facilitate the integration of these new adaptive 



-7- BEST COPY AVAILABLE 

strategies into an operational software system, it 
should be possible to develop and debug individual 
testing strategy programs as separate units and add 
them to the system with as little revision of the 
strategy program and sy s t em as possible . 

16. Item pools, particularly those used for research , 
constantly change. New items are added, old ones 
deleted, and items are constantly re-normed. Thus, 
it might be appropriate to frequently change item 
files to be used within a given testing strategy. 
Likewise, termination rules or criteria within an 
adaptive strategy, or other internal aspects of a 
strategy, might require occasional changes. In addi- 
tion, new item pools, measuring different kinds of 
abilities, might be added to a testing system, For 
all these reasons, data which might be frequently 
changed should be stored separately and be easily 
accessible so that changes can be easily made . 

17. Most time-shared computer systems can have only a 
limited number of programs in central memory at one 
time. Programs waiting to be executed are usually 
entered into central memory in inverse order of the 
amount of memory they require. Thus, an operational 
computerized ability testing system must use as little 
central memory space as possible to speed system re- 
sponse time and eliminate unnecessary delays for the 

t est ee • - 

18. To permit easy analysis of obtained data, the program 
system must store all obtained data in a form readily 
amenable to statistical analysis. At the same time, 
data must be stored as compactly as possible since 
data s t orage charges can add an apprec iable amount 

to the cost of computerized testing. 

One last feature is especially important to testing 
systems being used for applied purposes, although it has im- 
plications for research applications also: 

19. The system should permit file interpretation capabili- 
ties. Thus, at any time after the termination of test- 

ingr't h e~~p s y~c ho 1 ogi s t should ~b~e~a bTe to "fee all the test 

record of any individual and examine his performance 
in detail. This would include a reporting of test 
scores for the strategy or strategies administered 
to a given testee. Further, the system should permit 
a graphical display of the test performance on the 
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adaptive strategies. This display could, for example, 
chart the testee's progression through a pyramidal 
test structuic, through a flexilnvel test, or graph 
his performance on a Bayesian strategy test* Such 
displays would also be of great benefit in explaining 
test performance to testees in a de-briefing session 
or for didactic purposes in explaining adaptive test- 
ing to a variety of audiences , 



COMPUTERIZED ABILITY TESTING SYSTEM- -FILE 
AND PROGRAM STRUCTURE 

Computers work by manipulating stored data. Data can be 
stored by a computer in one of two ways; l) in the computer's 
central memory, which is a basic storage mode limited in storage 
capacity but with vfery rapid accessibility, and 2) on peripheral 
devices such as magnetic disc or tape which have almost un- 
limited storage capacity but relatively slower accessibility. 

F iles for Computerized Ability Testing 

A file represents a block of data which is stored on a 
magnetic disc, much as a file in the usual sense represents 
a block of information in a filing cabinet. 

The testing system works with two types of files: pro- 
gram files and data files. Program files are those which 
contain the machine language instructions for the different 
programs and subprograms used. They can be copied from a 
disc to central memory in a specified order, and then be 
executed. 

Data files contain information used by the testing system 
at various stages of testing. Some files are updated (i,e,, 
information is added to them) as they are used. The others 
are information (or reference) files only and as such can be 
easily modified as testing requirements change from time to 
time • 

Updated Files 

1 , Testing data . A file is saved for each testee, using 
his identification number as a . key for the file name. 
The details of this file structure are described in 
Appendix A, The first 300 characters (i.e., 30 10- 
character words) on this file contain demographic and 
summary information . Space is also allowed for some 
information necessary for the recovery feature. The 



ERLC 



-9- BEST COPY AVAILABLE 

remainder of the file contains information on each test 
item presented. This file is updated after a response 
is obtained to each item. 

2. Id en t if i cat ion number f ile . When a testee enters his 
identification number it is added to this file. Any 
other programs which refer to the individual testing 
data files can use this file to get a list of all 
subjects who have taken tests . 

Information File3 

1 . CATC ODE . Before a subject begins testing , a 3 to 7 
character keyword is entered by the proctor. This 
keyword determines each subject's testing assignments . 
The testing assignments controlled by the keyword in- 
clude type of testing strategy, specific item file, 
and type of feedback. File CATCODE contains a list 

of the valid keywords at any given time. To simplify 
the proctor ' s task, and to reduce errors in entering 
keywords, each keyword (i.e., combination of testing 
assignments and conditions) can be coded with a single 
letter. Thus, by entering a single letter, the proctor 
can designate the complete course of testing for any 
given testee. Appendix B explains current keyword 
characters • 

2. ABILS . Two of the testing strategies require an initial 
ability estimate. For retests it may be desirable to 
use the initial ability estimate from the first test. 
ABILS contains initial ability estimates, where taken, 
for all who took a test the first time and is referred 
to upon r et es ting. 

3. V0CAB4 . A random access file containing all of the 
vocabulary test items . Random access simply means 
that any test item can be directly accessed by item 
number as opposed to reading all items from the first 
to last sequentially. Random access files thus provide 
more rapid access to data than do sequential files. 
This file contains the vocabulary word to be defined, 
the five response choices and the correct response for 
each of about 600 vocabulary test items . Each item is 
identified by a unique item number, the "item reference 
number . " 

4. NUM5R . A random access file containing a number series 
item and correct answers for each of about 200 number 
series items. 



ERLC 



BEST COPY AVAILABLE 



5, MATRIX , A file used only by the Bayesian testing 

strategy, MATRIX contains item difficulty and discri- 
mination indices on the 328 selected vocabulary items 
used by the Bayesian strategy program. 



6, Item subset files * Each testing strategy program is 

written to work with sequential or working item numbers 
1 through N, where N is the maximum number of items 
which can be presented by that testing strategy, This 
simplifies programming by permitting' the programmers 
writing testing strategy programs to avoid dealing with 
actual item reference numbers. It also permits generality 
of testing strategy programs so that the same testing 
strategy can be used simultaneously with different item 
subsets. Prior to presenting a specific test form, a 
file is built up containing all of the item reference 
numbers to be used in that test in the order required 
by the branching paradigm. The file name consists of 
a mnemonic for a particular strategy followed by a 
single digit. This allows up to 9 different Item sub- 
sets for each strategy. The item subset number is 
entered with the keyword and can therefore be easily 
changed from testee to testee. At the time of item 
presentation, actual item reference numbers are assigned 
to each item in the testing strategy. 

For example, two-stage testing can administer up to hO 
items to any testee. The actual two-stage strategy 
program works only with working item numbers I to 40, 
A file TW0STG2 contains kO item reference numbers 
ranging from 1 to 600 which are entered into the file 
in the order required for items 1-hQ in the two-stage 
strategy program. Thus, the program might ask for item 
12, Before presenting the item, the reference item 
number of the 12 th item would be gotten from file TWOSTC?, 
e,g,, 283, and item number 283 would be obtained from 
file V0CAB4 and presented to the testee. 



Programs. Subprograms, and Overlays 

A computer program is a complete set of instructions for 
a specified task, A program has a number of functions. It 
must l) initialize any values used, or get them from a disc, 
tape, or other input device such as a CRT, 2) perform any cal- 
culations, and 3) cause any calculations yielding data that 
should be saved for later reference to be output in some re- 
trievable form. As a computer performs the instructions speci- 
fied in a program it is said to be running or executing that 
program. 
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In many instances, subsets of calculations or instruc- 
tions must be executed repeatedly, using different sets of 
data. This repeated set of instructions can be written as 
a subprogram (commonly called a subroutine). A subprogram 
consists of a set of instructions which is complete for some 
small task. The subroutine generally receives data from the 
main program and/or returns data to it. Subroutines can also 
input and output data. Thus, for example, instead of writing 
instructions to calculate a correlation coefficient at a 
number of different points in a program, the programmer would 
write one subprogram to perform those calculations. He would 
then refer to (or call) that subprogram whenever computation 
of a correlation coefficient was needed. 



A prograni and its associated subprograms represent a unit 
which must be stored in the computer's central memory during 
execution. Since central memory storage is at a premium, 
problems arise if a large program (i.e., one with a large 
number of instructions or stored data) is to be executed. To 
minimize this problem, overlays are used. An overlay is a 
program (with associated subprograms) which is stored on a 
peripheral storage device (e.g., magnetic disc) but, at the 
direction of another program, can be moved into memory and 
executed. When it is moved into memory, it replaces part 
of the program that was previously in memory. Thus a large 
program can be broken into units (overlays) which are moved 
into memory and executed at different times. Regardless of 
the size of the total program, the maximum central memory 
requirement is determined by the size of the largest combina- 
tion of overlays which will be used at one time. (The reader 
unfamiliar with the concept of overlays is referred to Appendix 
C for a more basic description of the overlay concept.) 

Overlay Structure for Computerized Ability Testing 

The computerized testing system is currently composed of 
a main overlay, 5 primary overlays, and 11 secondary overlays. 
A diagram of the overlay structure is shown in Figure 1. 



Main Overlay 

The main overlay is in memory at all times. Its primary 
function is to call the necessary primary overlays in the re- 
quired order, more than once if necessary. The main overlay 
also contains a large number of general purpose subprograms 
used by other overlays. These are mostly machine language 
subprograms obtained from the system program library which 
handle special arithmetic processes, and data input and out- 
put . 
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FIGURE 1, Overlay structure for the software system. 
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Initializing Overlay . This overlay reads in a keyword 
or a single-character keyword code which is verified by the 
CATCODE file, and breaks it into its components. These are 
checked for validity and coded in a convenient way for access 
by other parts of the program system (see Appendix B) , This 
overlay locates and then attaches the appropriate item number 
files. The maximum number of different items in the item 
subset, maximum number of items to be given (test length) and 
item numbers to be used are read by this overlay from the item 
number file for each test to be given to a specific testee. 

Instructional Overlay , This overlay presents instruc- 
tions and collects some demographic data. Date and time of 
beginning of testing are saved, A series of instructional 
screens on CRT usage are presented covering such topics as 
entering information, correcting errors, clearing trnns- 
mission line errors and entering a "?"• These screens, and 
a flow chart describing their structure, are shown in Appendix 
D, Most of these screens terminate with a question covering 
the material presented. The answer is checked, and if it is 
incorrect the screen is presented up to two additional times. 
If the testee does not answer correctly on the third attempt 
a bell is sounded to summon the proctor who can give the 
subject individualized assistance. 

Next, name and identification number are requested. If 
no identification number (either University or special number 
given by the proctor) is obtained, a social security number 
is requested. If a social security number cannot be obtained 
a bell is sounded and the subject is instructed to get a number 
from the proctor. If an initial ability estimate is needed, 
it is requested. If the test is a retest , the estimate will 
be read from the data file ABILS, After this, several 
screens containing instructions for either number series or 
vocabulary items are presented. 

When the instructional screens are completed, a data 
file containing all information obtained to that point is 
created and saved , The IDFILE containing testee identifica- 
tion numbers is updated and control returns to the main over- 
lay , 

A code to skip instructions (S) can be entered with the 
keyword. Only the identification number is then obtained. 
Likewise, the keyword can also contain a RETEST code (r) . 
In this case, the subject is given an opportunity to skip 
the instructions if he feels sufficiently comfortable with 
equipment usage. 
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The original instructions were developed for testing 
University students and were slightly modified for high- 
school students. The form of instructions to be used is de- 
termined by a program switch which is set by the particular 
procedure file used to initiate the testing program (See 
Appendix E for an explanation of procedure files). 

Testing Overlay . This overlay consists of a program 
PREITEM and a subprogram ITEM. 

PREITEM serves essentially as a routing program, as its 
main function is to call the appropriate testing strategy 
overlay. Before it does this, however, it executes a call 
to subroutine ITEM which instructs that subroutine to per- 
form certain initializing tasks in preparation for tept item 
presentation. After this is accomplished, the appropriate 
testing strategy overlay is called by program ITEM. Hie 
operation of the primary testing overlay is shown diagramma- 
tically in Figure 2. 

As Figure 2 shows, program PREITEM first execute- an 
initializing call to subroutine ITEM (A) . Once the initiali- 
zation operations have been completed, control returns to 
PREITEM (B) which then calls the appropriate testing strategy 
overlay (C). The testing strategy program then selects a 
working item number and calls the item presentation section 
of subroutine ITEM (d). After the item has been presented 
and a response received, control returns to the testing 
strategy program for selection of the next item (E) . The 
D and E cycle is repeated until the strategy program specifies 
termination of testing at which time control returns to 
PREITEM (F) and then the main overlay. 

ITEM is the core of the test administration system. It 
is responsible for presenting items, reading responses, and 
saving data. There are essentially two parts: initializing 
and item presentation. The initialization section is called 
from PREITEM. Data such as number of questions attempted 
and number correct are zeroed out. The time when actual 
testing starts is recorded. The appropriate item pool file 
(e.g., vocabulary or number series) is attached and prepared 
for random access read operations. 

The item presentation section is called from the testJng 
strategy overlay and is further divided into two sections. 
One section reads a vocabulary item from the disc, presents 
it, reads a response and checks it for validity and correct- 
ness. The second section performs the same operations but 
uses number series items. Since the two sections are functionally 
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FIGURE 2, Structure of the primary testing overlay 

AND ITS INTERACTION WITH THE SECONDARY 
STRATEGY OVERLAYS, 
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similar no further distinction will be made between them. 
Figure 3 illustrates the formats used by ITEM in presnitin^ 
vocabulary and number series items on the CRT. 

As Figure 2 shows % the item presentation section of ITEM 
is called from the individual testing strategy programs in 
the secondary overlay. That section of ITEM is given (as 
data) the working number of the it em to be presented . On 
receipt of this information, ITEM performs several operations. 
First, the testee's data file is updated to include the Last; 
item present ed as well as any test scores which might havo 
changed. The working item number is then converted to an 
actual item reference number and that item is read from a 
disc file. If, for any reason, that item is not available, an 
error return is given to the testing strategy program which 
can take appropriate steps. Otherwise, the item is presented 
and a response requested. The response is checked for vaMdjt 
If it is invalid, the testee is informed of that fact and the 
item is presented again. After the third invalid response to 
any item, a bell is sounded to summon the proctor 1 . 

If a valid response is obtained from the tester: it is 
checked for correctness (up to three alternative correct 
responses are checked for the free-answer number series 
items). If intermediate feedback (information on correct- 
ness/incorrectness of each item) is to be given, one of seven 
different appropriate feedback messages is presented in a 
pseudo-random order. Latency of the subject's response is 
calculated. The response code (right-wrong-?), actuaJ re- 
sponse, item reference number, number of times the item was 
presented, and the response latency are all added to the 
subject's data file before the next question is presented. 
The response code is returned to the testing strategy program. 
(See Appendix F for an explanation of response codes.) 

Secondary Overlays within the Primary Testing Overlay 

Tes ting strat egies . There are six secondary overlays 
associated with the primary testing overlay— one for each 
testing strategy. These programs are all written with general 
parameters to utilize any specified item subset; file. The 
programs select a working item number to be presented , and 
pass that number to subroutine ITEM. Subroutine ITEM locates 
and presents the appropriate item, processes the testee's 
response and returns a response code to the strategy program. 
The strategy program then chooses , in con junction with the 
item file it is using, the next item to be presented. A 
termination criterion is also checked to determine whether 
or not more items should be presented. 
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MAIL 

1. KNIGHT 

2. LIGHT ARM0R 

3. MEDIEVAL CLUB 

4. SHIELD 

5. LONG STAFF 

TYPE IN THE NUMBER 0F THE C0RRECT ANSWER AND PRESS THE RETURN KEY, 



(8 4 12 6 14 7 



FIGURE 3, Presentation screens for vocabulary and number serifs items. 
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Currently, any two testing strategies combined are limited 
to 500 unique items for potential selection and presentation. 
Thus, unless one of the testing strategies is the Bayesian 
strategy, no more than 500 items can be presented in a singLe 
testing situation, although the actual number presented to 
any testee is usually considerably less. The Bayesian testing 
strategy program draws from its own item pool of '}28 items 
and does not enter into the above combination. Thus, if the 
Bayesian strategy program is one of the two tests presented 
to a given testee a maximum of 828 items could be presented 
in one testing situation. 

Six testing strategy programs are currently operational 
on the secondary testing overlay (see Weiss, 197 ; * for more 
complete descriptions of these testing strategies): 

TWOSTG. The two-stage testing strategy presents an N^-item 
routing test followed by one of 3 or k N^-item 
measurement tests. The criteria for branching 
as well as the number of items on the routing 
(N 1 ) and measurement (Np) tests are variable. 
Currently a 10-item routing test and a '}0-item 
measurement test are being used (see Bet.z and 
"Weiss, 1973 for empirical results derived from 
two-stage testing) . 

FLEXLV. The flexilevel testing strategy uses an item 

pool with items arranged in oi"der of difficulty. 
A testee starts with the middle item, an item 
of average difficulty. A correct response 
leads to the next more difficult item; an in- 
correct response leads to the next easier item 
not previously presented. Currently a 79-item 
flexilevel structure, in which every testee 
is presented with 40 items, is. being used. The 
program is written to easily handle flexilevel 
designs of different numbers of items. 



LINEAR. Linear testing presents a fixed number of items 
in a fixed order (i.e., no branching). It is 
used for norming and for simulated paper and 
pencil testing. Currently '40-itetn linear tests 
are being used as the "conventional" tests , 
with which the various adaptive strategies can 
be compared (e.g., Betz and Weiss, 1973) • The 
linear tests used for item norming consist of 
varying numbers of items. 
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PYRAMD. The pyramidal testing strategy is based on a 
tree structure. All subjects begin with an 
item of average difficulty. The program selects 
a more difficult item upon receiving a correct 
response and a less difficult item following an 
incorrect response . The program is capable of 
administering a 2^4-stage pyramid (i.e., 2k items 
are administered to each testee). In addition, 
six different stepping patt erns are availble . 
With this feature, it is possible to skip the 
next more difficult or easier it em and select 
an item more difficult or easier than that one. 
This enables coarser adjustments or branching 
earl ier in the test and finer branching later 
after the testee's approximat e ability 1 evel 
has been reached. Additional variations in step 
size for pyramidal tests can be made by structur- 
ing specific pyramidal item subset files . 

STRTFD. The stradaptive testing strategy ( Weiss, 1973) 

uses an item pool with items divided into levels 
or strata based on their difficulty. Testees 
start at a level determined by a. prior ability 
estimate, which may be requested from the testee. 
After each response, the stratum from which a 
testee's items are chosen is examined to determine 
if he should be administered items from an easier 
or more difficult stratum. This decision is 
currently made only on the basis of whether or 
not the last response was correct, but a number 
of different criteria (latency, average latency 
at a level, number correct at a given level, 
etc.) can easily be included. In addition, 
differential step sizes are possible which branch 
the testee up or down by differing numbers of 
strata. This step size can be unique for each 
item. This variable step size enables larger 
branches to be made at the beginning of a test 
and smaller branches later in the test when an 
approximate ability level has been established. 
In its current form, the program can theoretically 
present 50 items at each of 25 strata. In its 
current applications the program can present a . 
maximum of 36 items at each of 9 strata. Because 
of item pool limitations not all strata have 36 
items and a total of 179 items can actually be 
presented . 
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BAYES . The Bayesian testing strategy has no pres true t ured 
item pool. Rather, the Bayesian strategy uses 
a special data file (MATRIX) which presently in- 
cludes information on the difficulty and dis- 
crimination of 328 test items selected from the 
larger vocabulary item pool. Items are chosen 
from this special pool using item difficulty 
and discrimination data in conjunction with 
estimates of the subject's prior ability based 
on items previously administered. An initial 
ability estimate is required and is used as a 
prior ability estimate to select the first item 
administered . Following administration of each 
item, a posterior ability estimate and its 
variance are c omputed . Testing t ermina t es 
after either a specified number of items have 
been given or a specified posterior vari ance 
has been re ached . 

Feedback Overlay 

The feedback overlay is a primary overlay which informs 
the subject of his score on the test or tests administered. 
If two tests are given and they use different types of items 
(e.g., verbal and numerical items), a separate score is given 
for each test. Unless post-feedback (p) for either test was 
specified by the proctor at the beginning of testing, program 
operation terminates at this point. If post-feedback was 
chosen for either test, the subject is given the opportunity 
to review those test items which he answered incorrectly. 
This review includes presentation of each incorrect item, the 
testee' s chosen response, and information on the correct re- 
sponse alternative (see Figure h) . Upon completion of the 
review, or if the testee elects not to review his performance 
in detail, control returns to the main overlay and the testee 
is informed that he is free to leave. 

Display Overlay 

The display overlay, a primary overlay, and its five nested 
secondary overlays, interpret a subject's data file and provide 
a graphic portrayal of his movement through the items comprising 
a test. These overlays can function in two nearly separate 
modes; l) As an integral part of the testing system, in which 
case it is called automatically when the testee has completed 
testing (both tests if two are given) or, 2) as a separately 
called overlay* 
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1. KNIGHT 
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THE C0RRECT ANSWER IS 2. 
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THE C0RRECT ANSWER IS 15 
Y0UR ANSWER WAS 123 

BITER "P M AND PRESS RETURN F0R THE NEXT ITEM. 



FIGURE 4, Feedback screens for vocabulary and number series items. 
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In the first mode (integral), the data file of tho testee 
who has just completed testing is interpreted and the testee 
(or the proctor) is given the option of destroying or saving 
the data file. The second mode (independent) is called by 
entering "PLOT" when a keyword is first asked for. The system 
then asks for the identification number or file name of a 
subject who completed testing at some time in the past. The 
testee's test results can be plotted and the user is given the 
option of reviewing other subject s f data files. In either 
mode, the test or tests the testee took are listed and the 
user is given the option of displaying either, both, or neither. 

Each testing strategy requires a different interpretation 
and plot so there are currently five seconday overlays, one 
for each strategy except linear testing. 

PI ot ting programs . These programs provide graphic por- 
trayals of a testee's movement through the item pool, separately 
for each testing strategy. They also provide scores for oach 
strategy. Sample plots produced by each of these programs arc 
illustrated in Appendix G. 

TWOSTG—plots testee performance on the two-stage adap- 
tive test. 

FLEXLV--pl ot s testee performance on the flexilevel adap- 
tive test. 

PYRAMD--plot s testee performance on the pyramidal adap- 
tive test. 

STRTFD--plots testee performance on the stradaptive test. 

BAYES--plots testee performance on the Bayesian adaptive 
test. 



Additional Sy s t em Features 

Recovery . The testing program is capable of recovering 
from interruptions caused at any point during testing either 
by the proctor or computer system failure of any kind. This 
is accomplished by updating each testee's permanent data file 
after each test item is presented. This update reflects in- 
formation on the last item presented as well as information 
on up to 2k key values which enable the exact item on which 
interruption occurred to be found. These key values vary 
between testing strategies, but in general include information 
such as the number of questions taken in different parts of the 
t est . The recovery procedure identifies the last item presented , 
i.e., that on which termination occurred, and that item is pre- 
sented again but is recorded as having been presented more than 
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once* To initiate recovery , the proctor enters RECOVER when 
the system requests a keyword. The testee's identification 
number is then requested. The subject's data file is then 
located, and searched to find the last item completed. The 
item on which interruption occurred is then presented again 
and testing continues from that point. Recovery, is possible 
immediately after abnormal termination occurs, or can be 
initiated at any future time, as long as the testee's data 
file is available on the disc. 

Program expansion . The individual testing strategies can 
be written and debugged as simple programs. The only inter- 
action they have with other overlay sections is in accepting 
a prior ability estimate if needed, requesting an item and 
accepting the response code for the answer. Once the strategy 
program is working properly, the addition of a few statements 
to it and to the main overlay permit it to be added to the 
overlay system. 

Data changes . Item subset files and keywords are the 
most frequently changed information. This information is 
stored on separate files which can be easily changed without 
requiring regeneration of the entire overlay system. 

Proctoring . Proctors need have no knowledge of computer 
programming. Once they have logged onto the computer (i.e., 
dialed the telephone and logged in by account number and 
password) two 2-word commands are necessary to load the program 
(see Appendix E) . The program then asks for a keyword. A 
keyword of up to seven characters or a single code letter can 
be entered. The test is completely self -administering after 
that point. The proctors must be able to aid subjects who have 
difficulty with the instructions or with entering their names 
or identification numbers. This aid requires only knowledge of 
the terminal hardware. Beyond that, the proctor must be able 
to recognize a variety of system problems that might occur 
(e.g., telephone line problems, computer instability, terminal 
failure) and initiate recovery procedures when necessary. 

Computer hardware and software requirements . 

Technical requirements (e.g., computer size and character- 
istics, transferability) of the software system are detailed 
in Appendix H. 
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APPENDIX A 

Description of Data Stored on each Testee's Data File 
Data is stored in 10-character words 
Word No. Data 

1 Identification number or Social Security number. 

2 Keyword as entered by proctor. 

3 Current location in program: 

0-1000 instructions 
1001-2000 Test 1 
2001-3000 Test 2, if given 
3001-^000 Post-Feedback 
k Elapsed time in seconds from time subject began 

instructions until testing was completed • 

5 Total time in seconds spent on instructions . 

6 Total time in seconds spent on test 1. 

7 Number of errors on instructional screens i-LO, 

1 character per screen. 

8 Number of errors on instruct i on ai screens LL-20 

1 character per screen. 

9 Number of items correct on test 1 • 
10-12 Testee's name, 30 characters. 

13 Characters 1-2: subject's estimated ability, 

if taken. 
Characters 3-8 : blank 

Characters 9-10: college code (01-27) 
1 k Social security number , if avail abl e . 

15 Date of testing. 

16 Seconds since midnight when testing began. 

17 Elapsed time in seconds spent on test 2. 

18 Maximum number of questions which could be 

given on test 1. 

19 Maximum number of questions which could be 

given on test 2 . 

20 Number of items attempted on test 1. 

21 Number of items attempted on test 2. 

22 First score on test 1. 

23 • First score on test 2 . 

2k (reserved for program for recovery information) 

25 Number of it ems correct on test 2 . 

26 Second score on test 1. 

27 Second score on test 2. 

28-30 (reserved for program for recovery information) 
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Structure of the tes ting data file (cont'd) 

Data on each vocabulary item is packed into one word 
as follows: 



character 1 : 



re sponse c ode 
c ode meaning 



0 
1 

2 



Item answered incorrectl} 
Item answered correctly 
Item answered with a ? 



2: 
3-6: 
7: 
8-10: 



actual response (l-5» ?=b) 
reference number of item pres en ted 
number of presentation s of screen 
response latency in seconds 



Data on each number series item is packed into 2 words 
as follows: 



Word 1: Character 1: 

2-3: 
4-10: 



Word 2: 



Character 1-4 
5-6 

7 

8-10: 



response code (same as vocabulary 

it ems ) 

blank 

actual response 

item reference number 
blank 

number of presentation s of 
screen 

latency in seconds 



In addition, for the Bayesian testing strategy, a prior 
ability estimate and prior variance estimate are calculated 
for each item and saved packed into one word as follows: 

characters 1-5 - prior ability estimate 
6-10 - prior variance estimate 

The length of each file is thus dependent upon the number 
of items given, the type of items given, and the testing 
strategy used . 
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Keyword Codes 

Before beginning testing, the testing program requests a 
keyword which must be entered by the proctor. This keyword 
completely describes the testing assignments for a testee. 
It consists of 3 to 7 characters composed of 1 or 2 three- 
character test keys plus an optional special instructions 
character. 



Each three- character test key has the following format : 



Character 1: Code letter for test 

A - St r ad apt ive 
B - Bayesian 
F - Flexilevel 
L - Linear 
T - Twostage 
V ~ Pyramid 

Character 2: Numeral 1 through 9 
nine potent ial item 
with the particular 
to be used • 



indicating whi ch of 
subsets associated 
testing s t ra t egy is 



Character 3s Feedback codes 
N - No feedback 

I - Intermediate feedback (i.e., feedback 

after each item) 
P - Postfeedback (i.e., feedback after 

c omp letion of testing) 



There are two special instruction characters: 

1) R: Retest - gives the testee the option of skipping 

instructions, if the test administration is 
a retest . 

2) S: Skip instructions (get the identification number 

only). This option is used mostly by staff 
in trial test situations. 

These two characters can be entered after a ^-character test 

code (between or after if two tests are given). No character 

is used for more than one function; each character position has 
its own unique code letters. 



ERLC 



-28- BEST COPY AVAILABLE 

Keyword Codes Icont ' d ) 
Examples of keywords: 

1. T3I— One test, a two-stage test (t) is to be given. 

The reference item numbers to be used will be 
read from a file TWOSTG^. Intermediate feed- 
back (i) will be given. 

2. Y5NL4PR or Y5NRL4P — Two tests, a pyramidal (Y) and 

linear (L) will be given. The pyramidal test 
will get item reference numbers from a file 
PYRAMDJj, and will present no feedback. The 
linear test wil 1 get item reference numbers 
from file LINEAR^ and will present post-feedback 
(P). The R designates that this is a retest 
situation and the testee should be given the 
option of reviewing instructions . 

Prior to presenting a series of tests, a file (CATCODE) 
can be prepared containing a list of all keywords which will 
be valid for a period of time. Each keyword has a single 
associated letter which can then be entered instead of the 
keyword. If the proctor enters a single letter when the key- 
word is requested, the program refers to this file and converts 
the entered letter to its associated keyword. The letters 
used to identify keywords are chosen to minimize both memory 
and typing errors by the proctors. 

In addition, there are three special cases where special 
commands other than a keyword may be entered: 



RECOVER: If this is entered after an abnormal termination 
of testing, the program will ask for the testee f s 
identification number, and will then recover 
available data and resume testing . 



SAMPLE: 



PLOT : 



This instructs the program to ask for a keyword 
again and to automatically call the display 
overlay upon completion of testing. 

This instructs the program to call the display 
overlay directly and to request information 
as to which testee's file should be displayed. 
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APPENDIX C 

An Explanation of Computer Program Overlays 

A comparison of computer programming with programmed 
learning texts might be useful in understanding the overlay 
concept. In programmed instruction texts the student usually 
proceeds through a series of frames. Each frame contains 
some substantive information, and the frames arc usually 
followed sequentially. However, some frames may contain 
questions which, depending on the student's answer, ma> 
branch him/her to a review section, to an expiana tion , or 
to new material. The process followed here is analagous to 
a computer executing a program; instructions in the program 
are executed sequentially, although differential branching 
can occur at a number of points in the program. 



For illustrative purposes assume that a programmed learn- 
ing text is designed to teach statistics. At several points 
in this text it might be appropriate to lead the student, step 
by step, through the calculation of the standard deviation of 
different sets of numbers. Rather than repeating the instruc- 
tional frames designed to lead the student through the cal- 
culational steps each time they are needed, it might be easier 
to have a section of the text called "Calculation of the Stan- 
dard Deviation". At any point in the text where it was necessary 
to calculate a standard deviation, the student would be helped 
to organize the set of numbers he was using at that point and 
then referred to the "Standard Deviation" section for calcula- 
tions. When the student finished the calculations he would be 
instructed to remember or record the value of the standard 
deviation and to return to the instructional frame which re- 
ferred him to that section. The section on the calculation of 
the standard deviation could then be used any number of times 
with only a few extra iris true t ions . This cal cul ational section 
in a programmed instruction text is directly comparable to a 
subprogram or subroutine in computer programming. 

Now, expand the programmed learning text concept into a 
series of texts for a complete course. Assume that at any 
given time a student is permitted to have in his possession a 
maximum of three texts— a main text, a primary text, and a 
secondary text. When the student first starts he is given the 
main text, as are all students. The main text might describe 
what primary and secondary texts are available to the student, 
give some examples of how to proceed through the text structure, 
determine where the student should start, and contain any fre- 
quently used or basic equations, calculations or tables. This 
main text might then refer the student to a primary text, for 
example a text on Bayesian statistics. The primary text would 
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have its own instructional frames, equations, tables, etc. The 
student would now have two texts. The primary text could refer 
to equations or tables that it contains itself or to the more 
general ones in the main text. If the student is interested, 
the primary text might refer him to one of a number of available 
secondary te"xts f for example one on discrete Bayesian models or 
one on continuous Bayesian models. The secondary text would also 
have instructional frames, tables, equations, etc. 

Figure 5 illustrates a possible text structure for the 
hypothetical programmed instructional sequence in statistics. 
The main text would give some basic information on statistics, 
instruct the student on use of the texts, aid the student in 
selecting which other texts he might want to use and contain 
some of the more basic, generally-used equations, formulas 
or tables , 

Four primary texts are available : Bayesian statistics, 
analysis of variance models, descriptive statistics, and 
correlational methods. Each primary text would contain sub- 
stantive information on its own special topic and general 
equations related to that topic and could aid the student in 
deciding whether or not he wanted to use any of the more 
specialized secondary texts. 

There are seven secondary texts, each one nested within a 
primary text. Discrete and continuous models are nested within 
Bayesian statistics. One-way, N-way and special models are 
nested within ANOVA models. There are no secondary texts 
associated with the primary text on descriptive statistics, 
Bivariate and multivariate correlational methods are nested 
within the correlational methods primary text. These secondary 
texts contain the most specific substantive information and 
most specific equations for various calculations. 

Figures 6a to 6f illustrate several possible configura- 
tions within the given text "system." A minimal configuration 
is illustrated in Figure 6a, A student who has just begun 
Would have this configuration. If he should decide to study 
the text on Bayesian statistics he would then get that text; 
this configuration is shown in 6b, Note that he still has the 
main text ; this enables him t o refer to any of the basic equa- 
tions in that text. If this student has no interest in either 
discrete or continuous Bayesian statistics he would not use 
either of the secondary texts nested within the Bayesian sta- 
tistics text • 

Figure 6c illustrates the next configuration. The primary 
text on Bayesian statistics has been returned and the text on 
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ANOVA models has been taken in its place. If the student is 
concerned only with N-way ANOVA models, he then obtains the 
secondary text (nested within ANOVA models) on N-way ANOVA 
models; he then has three integrated texts in his possession, 

In Figure 6d , both the primary text on ANOVA models and 
the secondary text on N-way ANOVA models have been returned 
and the primary text on descriptive statistics has been taken 
in their place. Note that this text does not have any associated 
secondary texta. There can be ito secondary text in use with 
this primary text. 

Figures 6e and 6f illustrate two possible configurations 
with the primary text covering correlational methods. In 6e, 
the secondary text on bivariate methods is in use. In 6f, the 
secondary text on multivariate correlational methods is in use. 
Note that there is no change in the primary text on correlation 
method s . 

As is shown in Figure 6, the student could have one, two, 
or three texts in his possession at any time. He must always 
have a main text (Fig. 6a) and can never have a secondary text 
without a primary text and a main text (Figs. 6c, 6e, and 6f) 
or a primary text without a main text (Fig. 6b and 6d). 

Since a main text does not necessarily require that a 
primary or secondary text be used (Fig. 6a), it should be 
understood that a main text cannot refer to any equations in 
a primary or secondary text. Likewise a primary text may not 
have any nested secondary texts (Figs. 6b and 6d) and cannot 
refer to any equations in a secondary text. A secondary text 
does require a primary and main text to be present (Figs. 6c, 
6e, 6f) and can therefore refer to equations in either one. 

The structures involved in these imaginary learning texts 
are directly analagous to those in an overlay system of compu- 
ter programs. Figure 7 illustrates an overlay configuration 
where a main, primary and secondary overlay are stored in 
central memory at one time. The arrows indicate the direction 
in which overlays themselves or subprograms within an overlap 
can be referenced or called. There is a main overlay which 
consists of a program and associated subprograms. It can refer 
to one of a number of primary overlays, each of which is another 
program with related subprograms. Each primary overlay can re- 
fer to any subprograms in the main overlay or to any of its own 
subprograms. It can also refer to its secondary overlays but 
to none of the subprograms in the secondary overlay. The 
secondary overlay can refer to subprograms in any of the three 
overlays within its vertical structure. 
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If the primary overlay calls (i.e., needs to execute) a 
new secondary overlay, the new one replaces the old one in 
central memory. Similarly, if the main overlay needs to 
execute a new primary overlay, the old primary overlay and 
any of its associated secondary overlays are moved out of 
central memory. The main overlay is always stored in central 
memory. A secondary overlay cannot be referred to by more than 
one primary overlay (i.e., secondary overlays are "nested" 
within the primary overlay) although it can be referenced 
any number of times by that primary overlay. 

The major advantage of the use of overlays is that very 
large computer programs can be sub-divided into sub-units which, 
under control of a master sub-unit (main overlay) can be moved 
in and out of central memory as needed. Thus, very large pro- 
grams can be executed with relatively small central memory 
requirements. This is quite important in time-shared computer 
systems which usually give higher program execution priority 
to those programs requiring small amounts of central memory. 
Another major advantage is that each overlay can be written 
and tested independent ly , greatly simplifying the programmer's 
task. The programmer developing a new overlay need not be con- 
cerned with the entire overlay structure but only with those 
sections directly related to the one he is developing. 
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Ins true t ional Screens 

There are currently 36 instructional screens. They are 
presented in this appendix in the order in which they are 
presented to a testee, Because there are a number of differ- 
ent testing conditions each screen is coded according to which 
conditions it represents. The codes follow: 

N--number series items 
V- -vocabulary it ems 
ER — error screen, presented if a previous screen was 
answered incorrectly 
R--retes t 

S--Instruc tions are to be skipped and only identifica- 
tion number requested 
HS--Screen for presentation to high-school students 
AB — Screens requesting ability estimate for stradaptive 
and Bay esian testing s trat egie s 
F--f eedback will be given 
NF--no feedback will be given 

Figure 8 illustrates the various paths that a single testee 
might follow through the instructional overlay. Each square 
represents an instructional screen and is numbered in accor- 
dance with the screens illustrated on the following pages and 
abbreviations listed above. 



There are five entry points. A is the standard entry 
point for a first time testee or a retest testee who chooses 
to review the instructions. B and C are entry points for 
testees who are to receive a retest (R specified on keyword). 
If these testees elect to review the instructions, they are 
routed to entry point A and exit through RETURN-H. If they 
elect not to review the instructions, screens 5-8 may be pre- 
sented to obtain name, identification number and ability esti- 
mate, and exit through RETURN-G. 

D is the entry point if S is specified on the keyword • 
Only identification number and ability estimate (if needed) 
are taken and the exit is through RETURN-G. 

E and F are intermediate entry points (between tests) for 
the situation where tests 1 and 2 use different forms of items. 
The testee is informed of the change in item type and given the 
option of going over the instructions for the new type. If he 
cheses to skip the instructions, he exits through RETURN-G and 
begins Test 2. If he elects not to skip the instructions, he 
is presented those instructions relating to the item form on 
Test 2 and exits through RETURN-H. 
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Th e wavy line (v) indicates a jump to error screens n^ER 
and n^ER. These two screens summon the proctor to aid a testee 
who is having difficulty with the instructional screens. n i ER 
is presented the first time the testee has difficulty; n^ER is 
presented if the proctor is called again. After these screens 
are presented the screen on which the testee encountered difficulty 
is presented again to be certain that the testee understands 
those instructions • 
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THIS TEST IS BEING QXVEN T0 YfU BY A CIMPUTER. 
THE INSTRUCTIONS FOR THE TEST WILL APPEAR 0N THIS SCREEN* 
YOU VI LL BE ASKED SOME QUESTIONS AT THE END OF EACH 
PART 0P THE INSTRUCTIONS^ T0 BE SURE THAT Y0U UNDERSTAND 
K0V T0 ANSWER THE TEST QUESTIONS* 

YOU MUST REM EMBER TWO THINGS IN ORDER TO TALK TO THE COMPUTER. 
1* DO NOT TYPE IN ANYTHING UNTIL A QUESTION MARK CT> 

APPEARS ON THE SCREEN* 
8* ONCE YOU HAVE TYPED IN AN ANSWER* THE COMPUTER 
DOES NOT RECEIVE IT UNTIL YOU PRESS THE 
••RETURN" KEY* 

NOV* THE FIRST THING YOU MUST DO IS FIND THE ••RETURN" KEY* 
THIS KEY IS AT THE RIGHT-HAND END OF THE SECOND ROW OF KEYS* 



VA1T FOR THE QUESTION MARK TO APPEAR* THEN PRESS THE "RETURN" 
KEY TO CONTINUE THE INSTRUCTIONS. 



Screen 1. First instructional screen 



YOU FOUND THE "RETURN" KEY* BUT YOU TYPEDl 



BEFORE YOU PRESSED IT* 

IN ORDER TO DO YOUR BEST ON THIS TEST* IT IS IMPORTANT THAT 
YOU F0LL0V INSTRUCTIONS CAREFULLY* 

NOV* PRESS THE "RETURN" KEY TO CONTINUE* 



Sci*«i«n I Kit. Scroeri presented if screen I .is in error 
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Y0U ARE ABOUT T0 TAKE A TEST DEALING WITH VOCABULARY ITEMS. 

SINCE Y0U HAVE WORKED WITH THE COMPUTER BEFORE* Y0U MAY N0T 
NEED T0 REVIEW THE INSTRUCTIONS. IF YOU HAVEN'T WORKED WITH 
VOCABULARY ITEMS BEFORE* IT MIGHT BE WISE TO GO OVER THE 
INSTRUCTIONS FOR VOCABULARY ITEMS. 

IF YOU WOULD LIKE TO GO OVER THE INSTRUCTIONS* ENTER "YES** AFTER 
THE QUESTION MARX APPEARS! IF NOT* ENTER "NO**. 



IN EITHER CASE* WAIT FOR THE QUESTION MARK* ENTER YOUR RESPONSE 
AND PRESS THE RETURN KEY. 



/ 



Screen i HV. First screen presented to test.ee taking ;i ret est for 
which the first test is vocnbulan items. If testee 
responds ^tS entire instruction.il series is presented 
beginning with I. If testee responds NO screens >H 
through 9 are presented. 



YOU ARE ABOUT TO TAKE A TEST DEALING WITH NUMERICAL ITEMS. 

SINCE YOU HAVE WORKED WITH THE COMPUTER BEFORE* YOU MAY NOT 
NEED TO REVIEW THE INSTRUCTIONS. IF YOU HAVEN'T WORKED WITH 
NUMERICAL ITEMS BEFORE* IT MIGHT BE WISE TO GO OVER THE 
INSTRUCTIONS FOR NUMERICAL ITEMS. 

IF YOU WOULD LIKE TO GO OVER THE INSTRUCTIONS* ENTER "YES" AFTER 
THE QUESTION MARX APPEARS* IF NOT* ENTER HO". 

IN EITHER CASE* WAIT FOR THE QUESTION MARX* ENTER YOUR RESPONSE 
AND PRESS THE RETURN KEY. 



Screen I UN. Similar to IHV except for number series items. 
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YOU SEEM TO BE HAVING TROUBLE WITH THE INSTRUCTIONS 
•R THE EQUIPMENT* PLEASE CALL THE TEST PROCTOR* HE WILL PERSONALLY 
HELP YfU LEARN TO OPERATE THE KEYBOARD* 



Screen n 1 ER . Error screen summoning proctor for the first, time. 

"It can be presented after any error screen is 
responded to incorrectly. After this screen is 
presented, the proctor can have the screen on which 
the error occured displayed again. 



YOU STILL SEEN TO BE HAVING TROUBLE WITH THE INSTRUCTIONS 

OR THE EQUIPMENT* PLEASE CALL THE TEST PROCTOR AGAIN. 

HE WILL PERSONALLY HELP YOU LEARN TO OPERATE THE KEYBOARD* 



Screen n^ER. A variation on screen n^R. This screen is presented 
instead of screen n 1 ER after that screen has been 
presented once. 
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/^^THE TEST Y0U ARE AB0UT T0 TAKE IS A MULTIPLE- CH0I CE TEST. 

AFTER EACH QUEST 1 0N , THERE IS A SERIES 0F S ANSWER CH0ICES* 
NUMBERED 1 THR0UGH 5. 

Y0U MUST CH00SE THE ANSWER THAT Y0U THINK IS C0RRECT* 
THEN TYPE IN THE NUMBER 0F THAT ANSWER. 

THE NUMBER KEYS ARE IN A BL0CK AT THE RIGHT SIDE 0F THE 
KEYB0ARD. THEY L00K LIKE THISt 



7 8 9 

4 S 6 

1 2 3 
0 



JUST FOR PRACTICE* TYPE IN THE NUMBER "S" WHEN THE QUESTION 
MARK APPEARS. REMEMBER T0 PRESS THE "RETURN" KEY AFTER Y0U 
HAVE TYPED IN THE NUMBER. 



Screen 2V. Second instructional screen for vocabulary items. 



Y0U DIDN'T TYPE IN THE NUMBER "5". 
HAVE AN0THER PRACTICE TRY. 
TYPE IN THE NUMBER -1 THIS TIME* THEN PRESS THE ••RETURN" KEY. 



Screen 2V ER . Error screen if screen 2V responded to incorrectly 
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IN THE TEST Y0U ARE AB0UT T0 TAKE* Y0U BUST TYPE Y0UR 
ANSWER USING THE KEYBOARD. TO TYPE NUMBERS AND FRACTIONS, 
Y0U WILL NEED T0 USE THE NUMBER KEYS* THE DECIMAL POINT* 
THE SLASH (/>* AND THS MINUS SIGN <-)• 

THE NUMBER KEYS AND THE DECIMAL P0INT ARE IN A BL0CK 

OP KEYS AT THE RIGHT SIDE OF THE KEYBOARD. THEY L00K LIKE THlSl 

7 8 9 

4 5 6 

I 2 3 
0 

THE SLASH (/> IS USED F0R FRACTIONS* SUCH AS 5/8. 
THE SLASH KEY (/> IS 0N THE B0TT0M R0V OF THE KEYBf ARD* 
THE SEC0ND KEY FR0M THE RIGHT. THE MINUS SIGN (-> IS 
THE THIRD KEY FROM THE RIGHT IN THE T0P R0V OF KEYS. 

JUST r0R PRACTICE* TYPE IN THE FRACTION 3/10 (THREE TENTHS) 
WHEN THE QUESTION MARK APPEARS. REMEMBER T0 PRESS THE 
"RETURN" KEY AFTER Y0U TYPE IN THE NUMBER. 



Screen L'N. Second instructional screen for number series items. 



THAT'S N0T QUITE RIGHT, F0R M0RE PRACTICE* TYPE IN 
FRACTION 7/10 (SEVEN TENTHS). 



Screen 7X ER. Error screen if screen 2N responded to incorrect 1 
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YOU'RE MING FINE SO FAR. Y0U KN0V H0V T0 TYPE IN YOUR ANSWER, 
AND Y0U KN0V THAT Y0U MUST PRESS THE "RETURN" KEY T0 SEND 
Y0UR ANSWER T0 THE COMPUTER* 



SUPP0SE Y0U MAKE A MISTAKE TYPING IN Y0UR ANSWER. Y0U CAN 
C0RRECT IT AT ANY TIME BEF0RE Y0U PRESS THE "RETURN" KEY. 

T0 MAKE CORRECTIONS, USE THE "BACK 3PACE" KEY, WHICH IS 

0N THE T0P R0W 0F THE KEYBOARD, THE SEC0NO KEY FR0M THE RIGHT. 

SUPP0SE Y0U TYPED IN 

5 

WHERE Y0U MEANT 4 



AS L0NG AS Y0U HAVE N0T PRESSED THE "RETURN" KEY Y0U CAN 
"BACKSPACE" THE BLINKING LIGHT T0 THE ERR0R, THIN TYPE 
IN THE REST 0F THE ANSWER CORRECTLY. 



T0 CORRECT THE MISTAKE ABOVE, "BACK SPACE" THE BLINKING 
LIGHT OVER THE "S", THEN TYPE IN "4" TO COMPLETE THE ANSWER. 

TRY IT WHEN THE QUESTION MARK APPEARS. 



Screen Third instructional screen. 



/ YOU APPARENTLY DIDN'T BACKSPACE QUITE RIGHT. HERE IS ANOTHER 

CHANCE TO PRACTICES 

4 

CORRECT THIS TO READ 5. THE SECRET IS TO BACKSPACE 
THE BLINKING LIGHT UNTIL IT COVERS THE CHARACTER YOU WANT 

TO CORRECT. THEN YOU MUST TYPE IN THE CORRECTION 

AND RE-TYPE THE REST OF THE ANSWER. 



GO AHEAD AND MAKE THE CORRECTION, WHEN THE QUESTION 
MARK APPEARS. 



Screen 3 V.H. 



Screen presented if screen '3 is answered incorrectly 
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NOW YOU KNflV WHAT T0 DO IN CASE YOU MAKE A MISTAKE. 

SOMETIMES THE C0MPUTER MAXES A MISTAKE* T00 (ALTHOUGH 

IT HATES T0 ADMIT IT>* AND Y0U CAN'T READ THE QUESTION 

0N THE SCREEN. IF THIS HAPPENS Y0U CAN REPEAT THE QUESTI0N 

BY PRESSING THE "RETURN** KEY (AFTER THE QUESTI0N MARK APPEARS) * 

WITHOUT TYPING ANYTHING* THAT SH0ULD S0LVE THE PROBLEM. 

FR0M N0V IN IF Y0U PRESS THE "RETURN" KEY VITH0UT TYPING 

ANYTHING ELSE FIRST* THE C0MPUTER WILL REPEAT THE 

SAME QUESTION. 

S0METIMES Y0U MAY N0T KN0V THE ANSWER T0 A QUESTION* AND WANT 
T0 SKIP IT. T0 D0 THIS* H0LD D0VN THE "SHIFT** KEY AND TYPE IN 
A QUEST! 0N MARK (T>. SINCE THE QUESTI0N MARK IS THE SAME KEY 
AS THE SLASH (/>* Y0U MUST H0LD THE "SHIFT" KEY D0VN WHILE 
Y0U PRESS THE 7. 

G0 AHEAD AND TYPE IN A QUEST 1 0N MARK. 
D0N'T FORGET T0 PRESS THE "RETURN** KEY AFTERWARD. 



Screen 'i . Fourth instructional screen. 



( Y0U DIDN'T TYPE IN A QUESTION MARK. 

REMEMBER* Y0U MUSTt 

1. HOLD DOWN THE SHIFT KEY* AND 

2. PRESS THE ? KEY. 

IF YOU DON'T HOLD THE SHIFT KEY DOWN WHILE YOU TYPE 
THE QUESTION MARK* THE COMPUTER READS A SLASH </) AND 
WILL TELL YOU TO TRY THE SAME QUESTION AGAIN. 

V NOW ONCE AGAIN TYPE IN A QUESTION MARK. 





Screen ! t KO, Screen presented if screen k is answered incorrectly 
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YflU ARB N0V ALMOST READY Tfl TAKE THE TEST. 

BY NflV YflU HAVE NAT ICED THE QUESTI0N MARK THAT APPEARS AT THE END flF 
BACK SCREEN. THE QUEST I AM MARK SIGNALS THAT IT*S YflUR TURN- 
BEF0RE THE QUEST I AN MARK APPEARS N0THINQ YAU TYPE 00 ES TA 
THE COMPUTER. SA BE SURE TA WAIT UNTIL YAU SEE THE QUESTIAN 
MARK BEFORE YAU TRY TA TYPE YAUR ANSWER. 

N0W WE NEED S0ME INFORMATION AB0UT Y0U. 

THE RESULTS AF THIS TEST WILL BE STRICTLY C0NFIDINTIAL. 

WE ARE INTERESTED IN Y0U AS PART 0F A LARGE GROUP* 

AND AT N0 TIME WILL YflUR TEST SC0RES BE CONNECTED WITH Y0UR NAME. 

BUT* WE NEED IDENTIFICATION S0 THAT WE CAN KEEP Y0UR TEST 
SEPARATE FRAN 0THER PEOPLE'S* AND SA THAT WE CAN CAMPARE THE 
RESULTS AF THIS TEST WITH OTHERS THAT YOU HAVE TAKEN. 

PLEASE TYPE IN YOUR FIRST NAME. 



Screen 5 • Fifth instructional screen. 



SINCE THIS IS A RET EST* WE NEED T0 HAVE SOME INFORMATION SO 
THAT WE CAN COMBINE THE RESULTS OF THIS TEST WITH THOSE OF 
THE ONE YOU TOOK EARLIER. 

PLEASE TYPE IN YOUR FIRST NAME. 




Screen 5R. Screen presented if retest is being taken, and 
testee did not request instructions. 



PLEASE TYPE IN YOUR MIDDLE INITIAL. 



Screen 6 . Sixth instructional screen, 



PLEASE TYPE IN YOUR LAST NAME. 



O reen 7. Seventh instructional screen. 
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N0V TYPE IN Y0UR SIX DIGIT 
IP YfU D0N'T KN0V IT* JUST 



UNIVERSITY ID NUMBER. 
TYPE IN A QUEST 1 0N MARK. 



Screen 8. Kiglith instructional screen, 



TYPE IN THE ID NUMBER GIVEN TO Y0U BY THE PR0CT0R. 



Screen 8 HS. Version of instructional screen 8, presented to high 
school students. 



SINCE Y0U D0N'T KN0V Y0UR I.D. NUMBER* PLEASE TYPE IN 
Y0UR S0CIAL SECURITY NUMBER. D0 N0T USE THE HYPHENS* JUST 
TYPE IN THE NINE DIGITS. 

If Y0U D0 N0T KN0V THIS NUMBER* TYPE IN A QUESTI0N MARK. 
REMEMBER T0 PRESS THE "RETURN" KEY. 



Screen 8. I Ett . Screen presented if subject does not. know .1.1). 

number in screen 8. 
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PLEASE SEE THE PROCTOR F0R A SPECIAL ID NUMBER. 



Screen 8.2 ER . Screen presented if testes knows neither 

nor Social Security number. 



THANK Y0U. 

LISTED BEL0V ARE SEVERAL OF THE COLLEGES WITHIN THE UNIVERSITY, 

1. COLLEGE 0r LIBERAL ARTS ( CLA) 

2. C0LLEQE 0F AGRICULTURE 

3. COLLEGE 0F BI0L0GI CAL SCIENCES 

4. SCH00L 0F BUSINESS ADMINISTRATION 

5. COLLEGE 0F EDUCATION 
6* GENERAL COLLEGE 

7. COLLEGE OF HOME ECONOMICS 
S. INSTITUTE OF TECHNOLOGY 
9. SCHOOL OF FORESTRY 

10. UNIVERSITY COLLEGE 

11. COLLEGE OF VETERINARY MEDICINE 

12. GRADUATE SCHOOL 

13. LAV SCHOOL 

14. — OTHER 



TYPE IN THE NUMBER OF THE COLLEGE IN WHICH YOU ARE ENROLLED* 
AND PRESS THE "RETURN" KEY. 



Screen 8.'} ER. Screen requesting student's col lege after social 

security number or proctor assigned number is 
obtained f o L I owing unsuccessful request for 1.1), 
number . 
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IN WHICH CATEQfRY IS YIUR CUMULATIVE OPA Tf DATE? 



3.7« 
3.51 
3*84 
3.01 
8.7.6 
2.51 
8.8.6 
8.01 
8.00 



Tf 4.00 
Tl 3.75 
Tf 3.50 
T0 3.S5 
Tt 3.00 
Tf 8*75 
Tf 8.50 
Tf 8.85 
fR LESS 



INTER THE CATEffRY C 1 THRfUOH 9 » AND PRESS THE RETURN KEY. 



Screen 8AB. 



Screen presented to university students requesting 
an ability estimate if a stradaptive or Bayesian 
test is to be given. 



EVERY Bf OT IS BETTER AT SfME THIN OS THAN fTHERSi 



CfMPARED Tf 0THER PEfPLE* HfW OffD Df YfU THINK YflUR VOCABULARY IS? 



BETTER THAN l 



V 



i hut fr 10 

8 fUT f r 10 

3 OUT f r 10 

4 out f r 10 

5 BUT f F 10 

6 fUT f r 10 

7 f UT f F 10 

8 BUT f F 10 

9 BUT fF 10 

TYPE IN THE NUMBER FRfM 1 Tf 9 THAT OIVES THE NUMBER BF PEBPLE 

Y1U ARE BETTER THAN (IN VOCABULARY). 



Screen 8AB HS, 



Screen presented to high school students requesting 
an ability estimate if a stradaptive or Bayesian 
test is to be given. 
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TH IS IS A MULTIPLE- CHS! CE TEST OF VtRO KNOWLEDGE. 

YOU WILL BE GIVEN A WORD* FOLLOWED BY FIVE OTHER WORDS 0R PHRASES. 
CH00SE FR0M AM0NG TH0SE FIVE* THE V0RD 0R PHRASE VH0SE 
MEANING IS CL0SEST T0 THAT 0F THE FIRST WORD. 

TYPE IN THE NUMBER CORRESPONDING T0 THE ANSWER Y0U THINK IS 
CORRECT. 

TYPE IN A QUESTION MARK IF YOU DON'T KNOW THE ANSWER. 
HERE IS AN EXAMPLE! 
DIFFICULT 

1. EASY 
&• HARD 

3. SOFT 

4. BIG 

5. SMALL 

TYPE IN THE NUMBER OF THE CORRECT ANSWER* THEN PRESS THE 
"RETURN" KEY. 



Screen '/V. Ninth instructional screen for vocnbulnrj items 



THE CORRECT ANSWER TO THE EXAMPLE QUESTION WAS NUMBER 
2 (TWO). "HARD" IS CLOSEST IN MEANING TO "DIFFICULT". 



NOW TRY THIS ON El 



CAR 



1. BOAT 
e. CAT 

3. TREE 

4. AUTOMOBILE 

5. SNOWMOBILE 



Screen VKR. Screen presented if screen ( >V was nnswured i 



ticiin'ci' 




LET'S TRY THE SAMPLE QUEST I W AGAIN. 
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THE ITEMS YfU ARE AMUT TO RECEIVE TEST NUMERICAL ABILITY. 

YfO WILL BE ASKED T0 RESPOND TO S0ME NUMBER SERIES ITEMS. 

EACH ITEM WILL CONSIST OF A SERIES OF NUMBERS. YfU ARE 
T0 PRf VIDE THE NEXT NUMBER IN THE SERIES. 

T0 CONTINUE, TYPE IN "NEXT" AND PRESS THE RETURN KEY. 



Screen '»N. Ninth instructional screen for numeric items. 

FIR EXAMPLE* YfU MIGHT GET THE FfLLfVING PROBLEM* 

15 16 17 10 19 80 81 

THE NEXT NUMBER IN THIS SERIES IS, OF COURSE, 88. 

Tf INDICATE Y0UR RESPONSE, TYPE IN THE NUMBER WIN 
THE 8UESTI0N MARK APPEARS. 

YOUR RESPONSE MAY HAVE UP Tf 7 CHARACTERS AND CAN 
CfNSIST fP NUMERALS (0-9) AND THE FfLLfVING CHARACTERS I 

/ • AND 

(NOTE THAT THE - IS Tf THE LEFT fP THE BACKSPACE KEY. > 

EXAMPLES f P CORRECT RESPONSES ARE BELOVi 

10 -311 11/90 .75 -1.38 

TO CONTINUE, TYPE IN "NEXT" AND PRESS THE RETURN KEY. 



Screen ION. Tenth instructional screen for numeric items, 

VHEN YOU HAVE DECIDED ON AN ANSWER, YOU MUST 
V* TYPE IN THE ANSWER, AND 
8. PRESS THE RETURN KEY. 



t 



BEFORE YOU PRESS THE RETURN KEY, YOU MAY CHANGE YOUR 
ANSWER BY BACKSPACING AND TYPING OVER IT. 

IF YOU CANNOT COME UP WITH AN ANSWER TO A PROBLEM, 
TYPE IN A T AND PRESS THE RETURN KEY. 

USE THE SCRATCH PAPER AND PENCIL PROVIDED FOR ANY 
COMPUTATIONS YOU NEED TO MAKE. 

TO BEGIN TESTING, TYPE IN "START 0 * AND PRESS THE RETURN KEY, 

O jreen 1 IN. Eleventh instructional screen for numeric items. 
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NOW VE VflULD LXKC TO GIVE YOU SOME VOCABULARY ITEMS. 

IP YOU ARE ALREADY FAMILIAR WITH HfV T0 V0RX WITH V0 CABULARY 
IT mi, Y0U MAY N0T NEED T0 00 THR0UGH THE SNSTRUCTI0NS. 

IF Y0U ARE N0T FAMILIAR WITH V0 CABULARY ITEMS J* Y0U SIMPLY 
WISM T0 REVIEW THE DIRECTIONS* ENTER •'YES- T0 REVIEW THEM* 
IF NOT, INTER "NO" TO SKIP THE DIRECTIONS. 

XM EITHER CASE* WAIT FOR THE QUESTION MARK* ENTER YOUR 
RESPONSE* AND PRESS THE RETURN KEY. 



Screen 1 .'.'V . 



Screen presented between tests where test 1 is 
number series and test .? is vocabulary. If.testee 
responds NO, testing resumes. If testee responds 
YES, screen <> is presented before testing resumes. 



NOW WE WOULD LIKE TO GIVE YOU SOME NUMERICAL ITEMS. 

IP YOU ARE ALREADY FAMILIAR WITH HOW TO WORK WITH NUMERICAL 
ITEMS* YOU MAY NOT NEED TO 00 THROUGH THE INSTRUCTIONS. 

IF YOU ARE NOT FAMILIAR WITH NUMERICAL ITEMS OR YOU SIMPLY 
WISH TO REVIEW THE DIRECTIONS, ENTER "TBS" TO REVIEW THEM. 
IF NOT* ENTER "NO" TO SKIP THE DIRECTIONS. 

IN EITHER CASE* WAIT FOR THE QUESTION MARK* ENTER YOUR 
RESPONSE* AND PRESS THE RETURN KEY. 



Screen 12N. 



Screen presented between tests where test I is 
vocabulary and test 2 is number series. If testee 
responds NO, testing continues, If testee responds 
YES, screens ION and LIN are presented before test i ng 
resumes . 
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THAT'S RIGHT. 

YfU MAY TAKE AS MUCH TIME AS YfU NEED Flit EACH GUESTlfN. 

AFT EM THE TEST* YfU WILL BE TfLD YfUR SCIRE* 

AND YfU WILL HAVE A CHANCE Tfl LffK AT THfSE YfU Of T VRfNQ. 

WIN YfU ARE HEADY Tf BEGIN* TYPE IN THE VfRD "START" AND 
AND PRESS THE -RETURN" KEY. 



Screen 10 F. 



lenth instructional screen if post-feedback is 
be presented. 



THAT'S RIGHT. 

NfV YfU ARE READY Tf BEGIN THE TEST. 

YfU MAY TAKE AS MUCH TIME AS YfU NEED FfR EACH QUESTIfN. 

AFTER THE TEST* YfU WILL BE TfLD Yf UR SC0RE. 

VHDf YfU ARE READY Tf BEGIN* TYPE IN THE VfRD "START" AND 
PRESS THE "RETURN" KEY. 



Screen t<> NF. 



Tenth instructional screen if no feedback is 
be Riven. 
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APPENDIX E 

Procedure File for Executing Testing Program 

Large computers are controlled by a software supervisory 
system (also referred to as a monitor, executive system or 
operating system) • This "system" is essentially a program which 
controls the flow of programs through the computer as effi- 
ciently as possible. The functions of this operating system 
include the allocation of memory space, the allocation and 
manipulation of files, the loading and execution of programs 
and the monitoring of time limits. 

There are two methods for giving commands to the Kronos 
operating system on Control Data Corporation's 6*400 computer: 
l) In an interactive mode where each command is entered on a 
terminal (CRT or teletype) and executed before the next command 
can be sent or 2 ) In a "batch" mode where a file (called a 
procedure file) containing an entire sequence of commands is 
prepared by the user and called at some later time: the user 
can issue no new commands until all of the commands on this 
procedure file have been executed. 

A combination of both of these types of commands is used 
to execute the testing program. After connecting the phone 
and logging on, the proctor must enter: 

OLD, CAT This attaches a procedure file. 

The system responds with a / (slash). The proctor then enters: 

CALL, CAT This command gives control to the proce- 

dure file CAT. 

The procedure file CAT consists of the 7 commands listed and 
explained below. 

RFL, 32500. This requests a field length or central 

memory storage space of 32500 words. 

GET , SUPTEST/UN=xxxxxxx 

This attaches a file called SUPTEST from 
account number xxxxxxx. The testing 
system currently operates on several 
different account numbers and program 
files are kept under one account number 
since duplicate copies are not needed. 
This procedure also protects the testing 
programs since they cannot be destroyed 
or altered by testees logged onto the 
system on different account numbers. 
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Procedure file for executing testing program (cont'd) 

OFPSW f 2. Switch 2 is an electronic switch which 

can be set on or off by commands and 
read by a program. Switch 2 is read 
by the testing program. If it is off, 
instructions developed for University 
students are presented. If it is on, 
instructions for high-school students 
are presented. This command turns 
switch 2 off. 



SUPTEST. The file SUPTEST contains the main over- 

lay for the testing system. This command 
causes the file to be loaded and executed, 
hence beginning the program. 

RETURN, SUPTEST, INPUT, OUTPUT, IN STR. This command is not 

executed until testing is finished • It 
serves the function of returning several 
of the files to the system; in essence, 
giving up the disc space they occupied. 

RETURN, TAPE7, TESTS, TAPE8. This command is similar to that 

above • 



EXIT. This returns control to the CRT or proctor 

who can determine if the testing program 
should be executed again. 

A second procedure file, CHS, is usable by entering 
OLD, CHS and CALL, CHS. 

This file is identical to the procedure file CAT except for the 
third command, which is 0NSW,2. instead of 0FFSW,2. This serves 
to turn on switch 2 which, when read by the testing program, 
causes instructions for high -school students to be presented. 
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APPENDIX F 

Response Codes Returned to Testing Strategy 
Programs by Subroutine ITEM 

3 - testing strategy program has requested a valid but 

unavailable item number 
2 - testing strategy program has requested an invalid 

item number 
1 - testing is terminated by proctor 

0 - item answered incorrectly 

1 - item answered correctly 

2 - item answered with a ? 
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APPENDIX G 

Displays of Test Results from all Testing 
Strategies except Linear Testing 



The display programs for Two- stage , Fl exile vel , 
Stradapt ive and Bayesian strategies were written 
by David Vale; the display program for Pyramidal 
testing was written by Robert . Swi sher . 
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REP0RT 0N BAYESI AN TEST 

ID NUMBER! 2067 DATE TESTED: 73/05/03 

X-C0RRECT O-INC0RRECT ?=N0 RESP0NSE 
ERR0R BAND PLOTTED IS ♦ AND - STANDARD DEVIATION 

ABILITY LEVEL ABILITY 
-2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 EST SD 

I 1 1 'i 1 1 1 1 1 I 1 1 1 



X. 

0. . 

....0.... 
...0. ... 
••••X«««« 



. ...0. . 

• • • Om • • 

• • • 0* • • 

> • • 0* • • 

Om* 
• ? • • • 

■ • X • • • 

• • X* • 

• • X • • 

• • X* • 



. 17 


1.41 


-.93 


.98 


-.42 


. 78 


-. 16 


.68 


-.42 


.61 


-.12 


.55 


-.29 


.51 


-.45 


. 48 


-.60 


. 46 


-.50 


. 44 


-.61 


.42 


-.72 


.41 


-.88 . 


. 39 


-1.01 


.37 


- 1. 16 


. 35 


-1.33 


.32 


-1.33 


. 32 


-1.25 


.32 


-1. 19 


. 30 


-1.13 


. 30 


-1.07 


. 28 



20 ITEMS WERE ADMINISTERED 
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RCPfKT ON PtCXILCVIL T£ST 
IDIflffCEftf 690134 DATE TrrttOi 73/07/12 




i « • • 

• •••• • 



LEAST MOST 
DIFFICULT DIFFICULT 
ITEM ITOf 

TOTAL MtflfBEn OF ITIJfS IN TEST* 79 
NI^ICR OF IT«TfS AOftNXfTIfl£0» 40 

NuwasB or i thus carreer* 2s 
4C«pe* as. so 
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REP0RT 0N PYRAMIDAL TEST 
ID NUMBER} 554911 DATE TESTED) 73/04/09 



STAGE 

1 2 3 4 5 6 7 8 9 10 1 1 12 13 14 15 

• • • • 

• • • • 



ir 



5+ 



• • • • 

• • • • 

1 2 3 4 5 6 7 8 9 10 1 1 12 13 14 15 
STAGE 



THERE WERE 11 CORRECT ANSWERS. 
THESE HAD A MEAN DIFFICULTY 0F .63 
THERE WERE 15 ANSWERS IN T0TAL. 
THESE HAD A MEAN DIFFICULTY 0F .75 
THE FINAL ITEM HAD A DIFFICULTY 0F 1.01 
THE DIFFICULTY 0F THE <N+1)TH ITEM IS 1.33 
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TIE PORT ON STRADAPTIVE TEST 
ID NUMBER! 469625594 DATE TESTED! 73/07/12 



STRATUM i 



(EASY) 
I 2 



PROP. CORRt 



LOO 1.00 
TOTAL PROPORTION CORRECT" .550 



CD! FFI CULT) 
8 9 







• 




• 




• 




• 




• 




• 








10*t£ 


• • 


• 




• 
• 


. • 2- 
13-^^ • 




• • 




>15- 


16+< 


• • 




J>17- 


18+^ 


• • 


• 
• 


"*19*_ . 
• ^**20- 



• 56 0. 00 



SCORES ON STRADAPTXVE TEST 

I. DIFFICULTY OF MOST DIFFICULT ITEM C 3 ERECT" . 1.49 
DIFFICULTY OF THE N* 1 TH ITEM- 1.44 

DIFFICULTY OF HIGHEST NON-CHANCE ITEM CORRECT" 1.49 



S 
6 
7 
8 

9 

10 
II 
12 

13 

14 

IS 



DIFFICULTY OF HIGHEST STRATUM 
VI TH A CORRECT ANSWER" 1.33 

DIFFICULTY OF THE N*l TH STRATUM- 1. 33 

DIFFICULTY OF HIGHEST NON-CHANCE STRATUM- U 33 

INTERPOLATED STRATUM DIFFICULTY- 1.37 

MEAN DIFFICULTY OF ALL CORRECT ITEMS- .88 



MEAN DIFFICULTY OF CORRECT ITEMS 
BETWEEN CEILING AND BASAL STRATA- 



i.28 



MEAN DIFFICULTY OF ITEMS CORRECT 

AT HIGHEST N ON* CHANCE STRATUM- I* 28 

SD OF ITEM DIFFICULTIES ENCOUNTERED- 



• 59 



SD OF DIFFICULTIES OF 
ITEMS ANSWERED CORRECTLY- 



1 46 



SD OF DIFFICULTIES OF ITEMS ANSWERED 
CORRECTLY BETWEEN CEILING AND BASAL STRATA- 
DIFFERENCE IN DIFFICULTIES 
BETCEEN CEILING AND BASAL STRATA- 1.36 

NUMBER OF STRATA BETWEEN 
CEILING AND BASAL STRATA- ' 1 



.18 
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REP0RT 0N TW0- STAGE TEST 
ID NUMBER! 2100 DATE TESTEDt 73/05/09 



M0ST DIFFICULT MEASUREMENT TEST 



R 
0 
U 
T 
I 
N 
G 



* .++- 



--++ +--+-- . ( 6) 



T 
E 
S 
T 



( 6 ) 



LEAST DIFFICULT MEASUREMENT TEST 



ABILITY ESTIMATE IN STANDARD SC0RE I St -2. 145 
ABILITY ESTIMATE IN PERCENTILE IS: 1.599 
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APPENDIX H 
Technical Requirements for a Computer- 
Assisted Testing System 

The current testing system was developed and is currently 
operational on MERITSS (Minnesota Educational Regional Inter- 
active Time-Sharing System). MERITSS is based on a Control 
Data 6k00 computer system which can handle more than JO ports 
simultaneously. Approximately 75 million characters of disc 
storage are available as well as approximately 65,000 words 
(10 characters or 60 bits per word) of central memory, making 
this a large-scale computer system. 

Disc or Central Memory Requirements 

The present testing system requires approximately 13,000 
words (130,000 characters) of central memory storage for execu- 
tion. Slightly more is required for compilation of the pro- 
grams involved --approximately 18,000 words (180,000 characters). 

Storage, on disc, of the different overlays used takes up 
approximately 26^,000 characters. The random access file for 
vocabulary items uses 119,000 characters of storage; that for 
number series items uses 6^,000 characters. In addition to 
these minimal requirements, each subject's data file takes up 
from 6k0 to 6000 characters of storage. Transferring the 
testees 1 data files to magnetic tape at frequent intervals will 
cut down total disc storage requirements but often over 200 
testee data files aire on the system representing as much as a 
million characters of disc space. The addition of other small 
auxiliary files such as CATCODE (for valid keywords) and ABILS 
(for ability estimates on retest) brings total disc capacity 
needed to approximately two million characters. 

Software Requirement s 

The entire testing system was written in Fortran and com- 
piled using the Control Data 6000 series Fortran IV compiler. 
This is a very powerful compiler containing almost all Fortran 
IV conventions . There are no system subroutines used in the 
testing system which are not siipplied by Control Data (i.e., 
there are no local installation routines used). 

Control Data 6000 series computers are based on a 60-bit 
( 10-charac t er ) fixed word length. This fixed word length was 
an important consideration in writing several of the routines 
and will seriously affect any transfer of the testing system 
to other computer installations. 
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Transportability 

The present testing system connects with user ports over 
standard voice grade telephone lines — one line per port. Thus, 
transferring the terminals to any location where phone lines 
are available is possible. 

Transf erabili ty 

Two aspects must be considered with respect to the trans- 
ferability of the testing system: i) use of CRT's from different 
manufacturers as terminals and 2) transfer of the program to 
another computer. 

CRT's , The current testing system was written for Computer 
Terminal Corporation (CTC) Datapoint 3000 CRT terminals. The 
testing program uses special codes* which are not standardized, 
to cause the CRT to move the cursor to the top of the screen, 
erase the screen and backspace the cursor. Use of other manu- 
facturers 1 CRT's might require changing these special codes 
within the testing program. 

The Datapoint CRT's have a screen which contains 2k 72- 
character lines. Other CRT's might have substantially smaller 
screens—usually fewer lines. Many of the instructional 
screens use the full 24 lines and would have to be rewritten 
for a smaller CRT. 

Programs . Transferring the testing system to another 
computer presents four levels of increasing difficulty of . 
modification. All but the last assume that a Fortran IV 
compiler is available and all assume sufficient disc storage 
space • 

1. The easiest transfer would be to a similar Control 
Data 6000 series computer. Some very minor modi- 
fications might be necessary to conform to local 
installation parameters but no major changes should 
be necessary; 

2 # A more difficult transfer assumes that the new system 
has sufficient central memory capacity but is not a 
Control Data 6000 series machine. The Control Data 
Fortran IV Series, compiler permits the use of star 
fields (*) in format statements for displaying strings 
of alphanumeric characters. This feature is not 
available with most other compilers so almost every 
displayed screen in the program would require re- 
writing in Hollerith fields. This would affect the 
instructional and display overlays significantly. 
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The second, and more severe, problem hinges on the 
fact that the CDC 6000 series computers use a 10- 
character (60-bit) word of fixed length. Some 
sections of the program are nearly independent of 
word length considerations. Subroutine ITEM, however, 
is very dependent on this fact. Program sections 
dealing with item retrieval from disc, item presen- 
tation and storage of data in the subject's data file 
would have to be rewritten. The instructional over- 
lay saves some collected data which is dependent on 
word length. The instructional screens themselves 
are nearly independent of word length. The feedback 
overlay's presentation section would require rewriting. 
The display overiaj' is somewhat dependent on word 
length but would probably not require major revision. 
In addition to these two major problems there are 
probably a number of Fortran statements unique to 
the CDC 6000 series Fortran IV Compiler. These would 
have to be found and changed to a form acceptable to 
the compiler at the new installation. 

The still more difficult transfer would be to a system 
lacking sufficient central memory storage to store the 
program in its present operating configuration. Assum- 
ing all necessary modifications described in (2) above 
have been completed, the difficulty of rewriting for 
less storage varies inversely with the afnount of central 
memory space available. Rewriting in this situation 
would require breaking the program into smaller and 
smaller overlays, segments, or chains until the largest 
configuration in use at one time would fit within the 
given central memory constraints. This is entirely 
feasible, although it must be noted that as the size 
of each overlay, segment, or chain decreases the number 
required increases and a significant amount of time 
would be used locating each one on disc and loading it 
into memory as needed. 

The most difficult transfer would be to a system where 
no Fortran IV compiler was available. In this situa- 
tion only the logic or structure of the system could 
be used and would serve as a structure around which a 
new system could be rewritten. 
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